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BROADCAST BROWSER INCLUDING MULTI -MEDIA TOOL OVERLAY 
AND METHOD OF PROVIDING A CONVERGED MULTI-MEDIA DISPLAY 
INCLUDING USER-ENHANCED DATA 



FIELD OF THE INVENTION 
[0001] The present invention relates generally to a system and 
method of creating and sharing enhancements to and in connection 
with a broadcast program to enhance the viewing experience of a 
number of viewers of the broadcast program. More particularly, 
the present invention concerns an overlay of multi-media tools 
which a system user may manipulate to create and engage in an 
interactive display with a broadcast program, and to transmit or 
otherwise provide the enhancements that he or she has created to 
additional users linked to the first user over a computer network, 
such as the Internet. 



BACKGROUND OF THE INVENTION 
[0002] Prior art systems are known which integrate television 
broadcasts with other video or audio content such as a stream of 
data broadcast over the internet. Although such merged displays 
may be interactive, they are preset in content and cannot be 
dynamically changed or dynamically created by the viewers of the 
broadcast nor shared with others. 



DESCRIPTION OF THE DRAWINGS 
[0003] The present invention will be better understood by reading 
the following detailed description, taken together with the 
drawings wherein: 

[0004] FIG. 1 is a schematic diagram of one exemplary system 
embodying the principles of the present invention, wherein 
multiple users view a broadcast program and simultaneously share 
information over a wide area network; 

[0005] FIG. 2 is a more detailed schematic diagram of each viewer 
display and manipulation system according to the present 
invention; 

[0006] FIG. 3 is a more detailed schematic diagram illustrating 
the inputs to a dynamic display controller of the present 
invention and an exemplary dynamically changed output; 

[0007] FIG. 4 is diagram showing the multiple layers that are 
displayed on a viewer display device; 

[0008] FIG. 5 shows a converged display including the multiple 
layers of FIG. 4, including a background layer for displaying a 
broadcast program and a user-prepared enhancement overlay layer; 

[0009] FIG. 6 is a schematic diagram of another exemplary system 
embodying the principles of the present invention, wherein 
multiple system users enhance a broadcast program via a set of 
multi-media tools provided by a Web server over the Internet; 
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[00010] FIG. 7 is another diagram showing the multiple layers that 
are displayed on a viewer display in the embodiment of FIG. 6; 

[00011] FIG. 8 shows a converged display including the multiple 
layers of FIG. 7, including a broadcast program (background) 
layer , a user-prepared enhancement overlay layer and a multi-media 
tool overlay layer; and 

[00012] FIG. 9 is a flow chart of one exemplary method of 
generating, providing and displaying user-prepared enhancements to 

M a plurality of viewers of a broadcast program. 

Q 

ffl DETAILED DESC RIPTION OF THE PREFERRED EMBODIMENT 

W 

iU [00013] A system 10, FIG. 1, on which the present invention can 

; be utilized and which embodies the present invention, includes a 

; a *f plurality of multi-media presentation systems (workstations) 12 

iriss; 

maintained by a plurality of system users or viewers. (The term 
1 user and viewer will be used interchangeably in the remainder of 
this description and should be construed to mean a person who 
perceives a broadcast program using his or her senses, including 
but not limited to sight and hearing.) The term multi-media 
presentation system is used herein to indicate a system capable or 
presenting at least video information to a user, although such 
systems generally also are capable of presenting audio information 
as well. However, the presentation of more than one media should 
not be construed as a limitation of the present invention. 
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Examples of such multi-media presentation systems 12 include 
personal computer (PC) systems, PC televisions (PCTVs) and the 
like. 

[00014] Each multi-media presentation system 12 includes a 
viewer computer 14, at least one display device 16, such as a 
monitor or television set, at least one audio output 18, such as 
one or more speaker that may be an internal component of a 
television set display device or provided as a separate speaker or 
multiple speakers. Each user multi-media presentation system 12 
also includes at least one input device 20, such as a keyboard, 
1^ mouse, digitizer pad, writing pad, microphone, camera or other 

m ... 

," I pointing or input generating device. 

gg [00015] As will be described more fully below, each multi-media 
presentation system 12 is provided with at least one broadcast 
U program signal 22, which may be provided in the form of broadcast 

q television programming (including cable and satellite television), 

M* 

closed circuit television, Internet web-TV or the like, received 
by means of a standard television broadcast signal over the air 
waves, cable television or satellite television, utilizing a tuner 
in each user computer 14. A broadcast program signal 22 may also 
be received from information stored on storage device 30 such as a 
hard drive, DVD, CD ROM, memory device or card, cassette tape, VCR 
tape, or other similar storage devices. In addition, each multi- 
media presentation system interfaces with a computer network 24, 
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which may be provided in the form of a local area network (LAN) , a 
wide area network (WAN) or a global computer network, such as the 
Internet . 

[00016] In one embodiment of the present invention illustrated 
and described herein for illustrative purposes only, the layout or 
arrangement of the network is in a peer-to-peer configuration. In 
yet another embodiment contemplated by the present invention, the 
present invention can be implemented in a configuration wherein a 
user or webmaster creates a web page, using the teachings of the 
present invention, for viewing by others. The web page would 
include the interactive content described herein. Third parties 

(generally unaffiliated with the interactive content creator) can 
then access the created web page and view/display the same in 
connection with a broadcast signal being displayed. In this 
embodiment, the unaffiliated third party viewer of the enhanced 
display need not even have the tools described herein. 

[00017] In yet another embodiment, the tools described herein 
may include only a limited set of tools or a tool set that creates 
interactive events and content only for the present view, and will 
not and cannot be sent to third parties. For example, a local 
user may be able to "click" on a display with a mouse and "throw a 
pie" onto a broadcast display at the location pointed to. 
Accordingly, the present invention may include only limited and/or 
local viewer controls. 
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[00018] The components of an example of a multi-media 
presentation system 12 are shown in FIG. 2. The heart of each such 
system is the user computer 14. Each user computer includes a 
central processing unit (CPU) 26, which controls the functions of 
the presentation system. The CPU interfaces a broadcast receiver 
28, which itself receives, as its input, the broadcast program 
signal 22. In one embodiment, the broadcast receiver 28 is a 
broadcast channel tuner which receives broadcast signals from a 
source such as a television broadcasting station or other 

□ 

CI programming provider or source. 

hi 

it 

!-fl [00019] Each user computer 14 also includes one or more 

UI 

ill internal storage devices 30, such as a disk drive, memory or CD 

i.n 

|h ROM where data, including overlays (as well as the broadcast 
□data), may be stored. A communications controller 32 is also 
HLl provided in each user computer 14, to control inputs received from 
I** and outputs transmitted to the other viewers via computer network 
24. The communications controller 32 may act as a second receiver 
for receiving a second data stream provided to the user computer 
over the computer network. 

[00020] In the preferred embodiment, the communications 
controller 32 may include a device such as a modem (for example, a 
telephone or cable modem) or a network interface card which 
receives information from a local or wide area network. 
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[00021] A dynamic display controller 34 (also referred to 
herein as a broadcast browser) is also provided with each user 
computer 14. The dynamic display controller interfaces the CPU 26, 
broadcast receiver 28 and communications controller 32 and 
receives, as input, the multiple data streams provided to the user 
computer by one or more of the broadcast program signal 22, the 
computer network 22 (via the communications controller 32) and the 
internal storage device 30. The dynamic display controller 34 
merges the multiple input signals and outputs a merged data signal 
□ to the display device 16. An audio processor 36 may also be 
UJ provided, as necessary, to receive audio data from the multiple 
US data sources and to provide the same to the audio output device (s) 

ru 

m is. 

[00022] In the preferred embodiment of the present invention, 
^ which is disclosed for illustrative purposes only and not 
^ considered a limitation of the present invention, the dynamic 

jsssss 

display controller 34 is implemented as computer software in the 
form of a browser user interface operating on the user computer 
14, which is a personal computer or individual computer 
workstation. Other embodiments contemplated include a client 
server configuration whereby a user computer 14 is connected to a 
server (not shown) which contains all or at least part of such 
computer software forming the dynamic display controller 34. 
[00023] Each multi-media presentation system 12 also includes 

7 

WAVE-004XX 

BOURQUE & ASSOCIATES, P. A. 



at least one input device 20, which allows a first user to direct 
input to the dynamic display controller 34 to control what is 
displayed on the display device 16, thereby allowing the user to 
control (i.e. generate) their viewing experience and in addition, 
to control the saving and/or displaying of the experience to the 
remaining users of the system 10, as will be explained in greater 
detail below. As can be seen more clearly from FIG. 3, each user 
computer CPU 26 receives, as a first input, a first data stream, 
such as a multi-media broadcast program signal 22 via broadcast 
receiver 28. It may also receive, as a second input, a data stream 
40 including one or more third party, user-prepared, enhancements 
to the broadcast signal input by a system user using one or more 
input device 20. Typically the user would interject images 
(video, hand drawn images, pictures, clip art, or the like), 
objects, audio (voice or other sound(s)) and/or text to be 
displayed on his or her display device 16. In this manner, a user 
can dynamically create a user experience in accordance with his or 
her personal preferences. As will become more fully apparent 
below, this user can also share his or her dynamically created 
user-prepared enhancements with other system users, to enhance 
their viewing experience or allow others to further modify and 
share their experience as well. 

[00024] As a third optional input, each user computer CPU may 
receive, via communications controller 32, a third data stream 42, 
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which is made up of shared enhancements to the broadcast program 
signal which were created by other users of the system and 
transmitted to the user computer over the computer network 24. 
The user computer CPU 26 merges the two or more data streams and 
provides a merged signal 44 to the display device 16. The CPU also 
provides, to communications controller 32, the data stream made up 
of the user-prepared enhancements, which the communications 
controller 32, in turn, transmits as a shared enhancement data 
stream 42' to the other users of the system. Alternatively, the 
enhanced data stream 42' may be stored on the user's internal 
storage device 30 for later replay or later transmission to 
others . 

[00025] As can be appreciated, using such a system, a user can 
enhance not only his or her viewing experience by preparing user- 
prepared enhancements, but he or she can also enhance the viewing 
experience of all users of the system by sharing his or her user- 
prepared enhancements to the other users of the system thereby 
creating a "community" viewing experience for connected/subscribed 
users . 

[00026] FIGS. 4 and 5 show how a layering or "overlay" strategy 
is utilized by the dynamic display controller 34 to control the 
display of the data provided by a broadcast signal and data 
representing user-prepared enhancements so that all of the data 
may be displayed in a single window or screen on each display 
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device 16. The dynamic display controller displays, in a 
"background" layer 50, the broadcast signal. Then, an overlay is 
displayed in the same window in at least one additional layer 54 
on top of the background layer 50. (It is understood that the 
order or layers can be reversed, if desired.) In order to allow 
the broadcast signal in the background layer 50 to be visible 
through the second layer 54, the second layer utilizes a 
substantially transparent background 56 or, as is disclosed 
herein, a background called or named "broadcast" to signify the 
source of the background information. 

[00027] One method of creating an "overlay" is to display the 
second or overlay layer as a "partial" layer. For example, every 
third pixel might be turned on or used. This would leave two- 
thirds of the pixels through which the background image could 
"bleed through". Another technique would be to resize the 
broadcast image to, for example, four-fifths (4/5' s) of its normal 
size, and place the interactive information and/or tools in the 
margins or otherwise in the remainder of the one-fifth (1/5) 
viewing image not utilized for the broadcast signal. Other 
methods may also be utilized without departing from the scope of 
the present invention. 

[00028] At least one of the additional layers 54 includes a one 
or more of user-selectable multi-media tools 56, which may be 
provided in the form of a toolbar 58. Of course, the toolbar 58 
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may be re-positioned to any portion of the screen as the user 
desires as is well known in the art. The user-selectable tools 56 
allow a user to manipulate the overlay to modify the layers 
displayed on his or her display device. 

[00029] Examples of user-selectable tools include drawing tools 
that allow a user to reference or comment on one or more objects 
appearing in the underlying broadcast signal on the background 
layer of the display. Such drawing tools may include lines, 
arrows, text boxes, thought bubbles, speech bubbles and the like. 
The user-selectable tools may also include one or more graphic 
insertion tools, which are responsive to a user input, to insert a 
graphic (image, picture, drawing, video clip, etc.) obtained from 
a graphic library into the overlay being displayed in the 
additional layer 54. Such graphics libraries may be stored in 
internal storage provided by the user computer or may be stored in 
remote databases, which are accessible via the computer network. 
Other examples include a magnifying glass, circle, square of other 
geometric shape , animated or non-animated character, escaping gas 
bubble, burst, explosion, highlighting or in short, anything 
object, shape, device or action that references the underlining 
broadcast programming. 

[00030] The user-selectable multi-media tools may also include 
an audio device to receive, store, edit and/or otherwise provide 
user-prepared auditory enhancements to the broadcast program. Of 
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course, like the video signals transmitted to the other users f 
user-prepared auditory enhancements can also be transmitted to the 
additional system users over the computer network where they would 
be output on audio output devices included at each user's multi- 
media presentation system. 

[00031] In addition to the possible text, graphic and audio 
tools, the tools may also include a user-selectable delivery icon, 
which can be used by the user to trigger the delivery of any user- 
prepared enhancements to those of the plurality of additional 
system users who are included on a delivery list maintained by the 

'St? 

0 

hj user of the system that has created the user-prepared 

i 

uj enhancements. Of course, only those additional system users that 

;"H 3 

:.|| are logged onto their system and viewing the same underlying 
broadcast program as the user creating the enhancements will be 

U able to display or otherwise output the shared enhancements on 

i.f] 

O their display or audio output devices however, the user created 

jiiieS* 

enhanced broadcast may be stored on a storage device of another 
user for viewing at la later time by the user. 

[00032] When the multiple data streams are merged, the 
resulting display appearing on the display device may appear in a 
single window 60, where the user-prepared enhancements will 
directly coincide with the portions of the underlying broadcast 
data stream to which they are directed if the user creating the 
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enhancement creates and sends/stores them as they coincide with 
the broadcast signal. 

[00033] For example, speech bubbles 62 or thought bubbles 64 
can be positioned adjacent a character 66 to which the speech or 
thought is to be attributed, text or speech inserted, and then 
transmitted (such as by hitting the return key or clicking the 
'"mouse" button) or stored such that the respective alignment of 
the enhancements with the broadcast signal is maintained. Text 
boxes 68 may be positioned where they will minimize interference 
with important objects appearing in the underlying broadcast. 

[00034] In another embodiment, the user may simply view only 
another's parties interactive information or overlay. In yet 
another embodiment, the interactive portion and the broadcast 
signal portion of the ultimate display may be merged prior to 
their arriving at the user's viewing device and therefore would be 
played out and viewed as one stream of information containing both 
the broadcast signal information and the interactive signal 
portion. 

[00035] FIGS. 6-8 show an alternative embodiment of a system 10 
for communicating between a plurality of multi-media presentation 
participants. In this embodiment, each user multi-media 

presentation system 12 interfaces with a Web server 7 0 via the 
Internet 72. The Web server 70 provides a multi-media tool 
overlay 74 as well as a user-prepared enhancement overlay 76. 
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[00036] Each user multi-media presentation system 12 is similar 
to those described above with respect to the embodiment of FIGS . 1 
and 2. However, instead of storing a multi-media tool overlay in 
local system memory and having the dynamic display controller 
retrieve the overlay from the system memory, in this embodiment, 
each user computer accesses the web server 70, where the overlay 
information is maintained. Nonetheless, each user computer would 
still include a dynamic display controller 34 for merging the 
overlay information accessed and manipulated via the web server 
with the broadcast presentation 22 received directly by each user 
system. 

[00037] In this embodiment, since multiple users will access a 
common multi-media tool overlay 74, a display strategy utilizing 
three or more layers may be utilized. In this manner, each system 
user can access the same tool overlay and use the tool overlay to 
create and store user-prepared enhancements to the broadcast 
signal that are stored on a third display layer 52. Each user 
will have a unique third display layer 52, which may also be 
referred to as a user-prepared enhancement overlay. While there 
will be a common multi-media tools overlay, each user will create 
his or her own user-prepared enhancement overlay. 

[00038] The user-prepared enhancement overlay will then be 
transmitted to the other users of the system in a manner similar 
to that described above with respect to the self-contained, peer- 
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to-peer system of FIGS. 1 and 2. Once the layers are merged by 
the dynamic display controller, the use of transparent backgrounds 
on the each overlay layer will allow the display to appear as if 
the user-prepared enhancements were simply inserted into the 
underlying broadcast as is shown in FIG. 8. 

[00039] In order to emphasize user-prepared enhancements, a 
special tool may be provided with the plurality of multi-media 
tools. This tool will be referred to as a "broadcast mute'' tool. 
The purpose of the broadcast mute "tool" is to dampen or otherwise 
minimize the interference of the underlying broadcast signal so 
that the user-prepared enhancement overlay appear more prominently 
in the merged display. One means by which the broadcast mute 
feature may emphasize the user-prepared enhancement overlay is to 
provide a video mute feature. The video mute feature may be 
implemented as a control for the brightness and/or contrast 
and/or hue signal of the underlying broadcast signal sent to the 
display device. By lowering either or both of the brightness or 
contrast signal to the display device, the appearance of the 
broadcast data in the merged display will be dampened so that the 
user-prepared enhancements will be more prominent. Another 
embodiment contemplates that another layer may be inserted between 
the broadcast signal layer and the interactive layer softening the 
background providing for more contrast to the interactive layer. 
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[00040] Since the purpose of the broadcast mute tool is to 
provide emphasis to the user-prepared enhancements, when such 
enhancements are provided to the remainder of the users as shared 
enhancements, selection of the broadcast mute tool will affect the 
underlying broadcast signal of all users to whom the enhancement 
is shared. 

[00041] In a similar manner as the broadcast mute tool, the 
tool set 58 may also include an audio mute tool. The audio mute 
tool will operate generally in a similar manner to the video mute 
tool. However, instead of affecting the underlying broadcast's 
video signal, it would allow audio enhancements to be highlighted 
by reducing the volume of the underlying broadcast signal. Of 
course both the video mute and audio mute features could be used 
together. 

[00042] A method of generating and providing user-prepared 
enhancements to a plurality of viewers of a broadcast program 100 
is shown in FIG. 9. To utilize the method, a plurality of viewers 
of the broadcast program will utilize a display device for viewing 
the broadcast program. Each viewer will also have a computer for 
controlling the display device and for interfacing each user to 
the other viewers over a computer network. 

[00043] The method 100 begins by displaying a broadcast program 
in a background layer on at least one viewer display device, act 
110. Next, at least one overlay layer is provided on each viewer 
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display device, act 120. Each overlay layer includes a 
transparent background to allow the broadcast program being 
displayed on the background layer to "bleed through". At least 
one of the overlay layers includes a plurality of user selectable 
multi-media tools, which are responsive to user input, for 
manipulating at least one overlay layer by including user-prepared 
enhancements thereupon. 

[00044] Then, user interaction with the provided multi-media 
tools is monitored and any user-prepared enhancements input by a 
viewer using the tools is stored, act 130. The user-prepared 
enhancements are then transmitted to any additional users of the 
system who are viewing the underlying broadcast presentation, act 
140. Preferably, the user-prepared enhancements are transmitted 
in response to a user selectable delivery icon so that the user 
can complete the user-prepared enhancement and then deliver the 
enhancement when he or she so desires and to whom he or she 
desires . 

[00045] Finally, in act 150, the user-prepared enhancement that 
has been transmitted to the additional system users is displayed 
on at least one overlay layer on top of the broadcast layer being 
displayed on a display device at a receiving user's system. 

[00046] Accordingly, the system and method described above, 
which embody the present invention, allows viewers of a broadcast 
presentation to enhance their own viewing experience and enhance 
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the viewing experience of others by preparing and sharing multi- 
media enhancements to the underlying broadcast presentation. 
[00047] Modifications and substitutions by one of ordinary 
skill in the art are considered to be within the scope of the 
present invention, which is not to be limited except by the claims 
which follow. 
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